Systems and methods for evaluating difficulty of spoken text

ABSTRACT

Systems and methods are provided for assigning a difficulty score to a speech sample. Speech recognition is performed on a digitized version of the speech sample using an acoustic model to generate word hypotheses for the speech sample. Time alignment is performed between the speech sample and the word hypotheses to associate the word hypotheses with corresponding sounds of the speech sample. A first difficulty measure is determined based on the word hypotheses, and a second difficulty measure is determined based on acoustic features of the speech sample. A difficulty score for the speech sample is generated based on the first difficulty measure and the second difficulty measure.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority from U.S. ProvisionalApplication Ser. No. 61/727,276 entitled “An Automated Spoken TextDifficulty Evaluation Method,” filed 16 Nov. 2012, the entirety of whichis hereby incorporated by reference.

FIELD

This disclosure is related generally to audio processing and moreparticularly to spoken text difficulty estimation.

BACKGROUND

The use of audio in teaching and examination can be highly beneficial.For example, the use of audio of a person speaking can be useful indetermining an examinee's level of comprehension. Audio listening itemscan also be useful in helping a student improve certain skills such aslanguage learning. The benefit of such audio of speech samples can besignificantly diminished when the difficulty of the speech sample in theaudio is substantially mismatched with a listener's ability level (e.g.,a novice language learner may struggle to understand a native,fast-talking speaker of an unfamiliar language).

SUMMARY

Systems and methods are provided for assigning a difficulty score to aspeech sample. Speech recognition is performed on a digitized version ofthe speech sample using an acoustic model to generate word hypothesesfor the speech sample. Time alignment is performed between the speechsample and the word hypotheses to associate the word hypotheses withcorresponding sounds of the speech sample. A first difficulty measure isdetermined based on the word hypotheses, and a second difficulty measureis determined based on acoustic features of the speech sample. Adifficulty score for the speech sample is generated based on the firstdifficulty measure and the second difficulty measure.

As another example, a computer-implemented system for assigning adifficulty score to a speech sample includes a computer-readable mediumconfigured to store a digitized version of a speech sample. An automaticspeech recognizer is configured to perform speech recognition on thedigitized version of the speech sample using an acoustic model togenerate word hypotheses for the speech sample and to perform timealignment between the speech sample and the word hypotheses to associatethe word hypotheses with corresponding sounds of the speech sample. Atextual difficulty determination engine is configured to determine afirst difficulty measure based on the word hypotheses for the speechsample. An acoustic difficulty determination engine is configured todetermine a second difficulty measure based on acoustic features of thespeech sample. A difficulty score calculator is configured to generate adifficulty score for the speech sample based on the first difficultymeasure and the second difficulty measure, and a computer-readablemedium is configured to store the difficulty score.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting a computer-implemented system forassigning a difficulty score to a speech sample.

FIG. 2 is a block diagram depicting generation of a difficulty measure.

FIG. 3 is a table depicting twelve example difficulty measures.

FIG. 4 is a block diagram depicting example contexts for usage of adifficulty score.

FIG. 5 is a flow diagram depicting a computer-implemented method ofassigning a difficulty score to a speech sample.

FIGS. 6A, 6B, and 6C depict example systems for use in implementing aspeech sample scoring engine.

DETAILED DESCRIPTION

FIG. 1 is a block diagram depicting a computer-implemented system forassigning a difficulty score to a speech sample. A computer-readablemedium 102 stores a repository of speech samples for evaluation. Suchspeech samples can come from a variety of sources, such as speechsamples that are specifically recorded for use with a system or reallife speech samples, such as a recording of a news broadcast. A speechsample 102 is provided from the repository to a speech recognizer 104for analysis. The speech recognizer 104, such as an automatic speechrecognizer that includes a trained acoustic model or a humantranscriber, identifies word hypotheses of words that are thought to bepresent in the speech sample. The speech recognizer 104 further performsa time alignment between the speech sample 102 and the word hypothesesto associate the word hypotheses with corresponding sounds (phonemes) ofthe speech sample (e.g., the word “dragon” is used at 0:30.45-0:30.90 ofthe speech sample). The word hypotheses and time stamp associations 106are provided to a speech sample scoring engine 108 for furtherprocessing and generation of a difficulty score 110 for the speechsample 102. The difficulty score 110 is provided to the repository thatstores the speech sample 102 or another computer-readable storage mediumfor storage and subsequent access.

FIG. 2 is a block diagram depicting generation of a difficulty measure.A speech sample 202 is accessed and provided to an automatic speechrecognizer 204 that generates word hypotheses for the speech sample 202and time stamp associations for those word hypotheses that are output206 to a speech sample scoring engine 208. The speech sample scoringengine 208 generates a plurality of difficulty measures 210, 212 thatare provided to a scoring model 214 for generation of a difficulty score216 that is associated with a speech sample 202 under consideration.

The plurality of difficulty measures 210, 212 may be determined based ona variety of characteristics of the speech sample 202 underconsideration. Certain difficulty measures are based on the content ofthe speech sample (i.e., the words that are present or thought to bepresent in the speech sample as represented in the transcript of wordhypotheses 206). Text feature analysis is performed at 218 to determinea first difficulty measure 210 based on the word hypotheses 206 for thespeech sample 202. For example, in one embodiment, the first difficultymeasure 210 represents the difficulty of vocabulary used in the speechsample 202, where the word hypotheses are compared to a vocabularydifficulty repository that identifies a difficulty associated with eachword. A total vocabulary difficulty (e.g., an average difficulty) isoutput by the text feature analysis 218 for use by the scoring model214.

As another example, certain difficulty measures may be based on pureacoustic characteristics of the speech sample. A speech sample 202 isprovided to the speech sample scoring engine 208 for acoustic featureanalysis at 220 for determination of a second difficulty measure 212. Inone example, a pure acoustic characteristic is determined by analyzing anumber of pauses in the speech sample 202 to determine fluencydifficulty measures such as silences per unit time or silences per word.Such a second difficulty measure 212 is provided to the scoring model214 for generation of a difficulty score 216 representative of thedifficulty of the speech sample.

In a further example, certain difficulty measures are based on bothacoustic and textual characteristics of the speech sample. For example,a pronunciation difficulty measure measures a correctness ofpronunciation of words in the speech sample. For each word hypothesis206 provided to the speech sample scoring engine 208, a properpronunciation(s) is accessed from a dictionary repository. Thatcannonical pronunciation is compared to an actual pronunciationexhibited in the speech sample 202 at 220 to determine a quality ofpronunciation in the speech sample, which is output from the acousticfeature analysis 220 to the scoring model 214 as the second difficultymeasure 212.

The scoring model 214 receives one or more difficulty measures 210, 212and generates a difficulty score for the speech sample 202 based on thereceived difficulty measures 210, 212. In one example, a number ofdifficulty measures are analyzed using linear regression to identify acorrelation between those difficulty measures and speech sampledifficulty (e.g., difficulty measured by a human scorer). A subset ofdifficulty measures which correlate significantly to the speech sampledifficulty are selected, and a weighted average of those selecteddifficulty measures 210, 212 is used to generate a difficulty score 216for speech samples 202.

A speech sample scoring engine can be configured to utilize a variety ofdifficulty measures in generating a difficulty score for a speechsample. FIG. 3 is a table depicting twelve example difficulty measures.The table includes a wpsec measure that identifies the number of wordsspoken per second in a speech sample; a longpfreq metric that identifiesa number of long silences per word (e.g., greater than 0.495 seconds); awdpchkmeandev metric that identifies an average chunk length in words,where a chunk is a segment whose boundaries are set by silences ordisfluencies over a threshold length (e.g., 0.2 seconds); a stretimemeanmeasure that identifies a mean distance between stressed syllables inseconds; a phn_shift metric that identifies the mean of absolutedifference between the normalized vowel durations compared to standardnormalized vowel durations estimated on a native speech corpus; anouncollocationspersent metric that identifies a number of nouncollocations per clause; a type_token_ratio metric that identifies anumber of types divided by a number of tokens; a voc40_and_wds_not_tasametric that identifies a normalized count of word types whose TASASFI<40 or words which are not found in TASA; a listenability_biber_typemetric that identifies the average frequency of word types in theresponse; an avg_sent_wrd_cnt grammar measure that identifies an averagenumber of words in a sentence; a long_sentences metric that identifiesthe number of sentences that contain more than 25 words; asent_per_1000words grammar metric that measures a number of sentencesper 1000 words in the speech sample; as well as others. Other measuresinclude, a number of silences per word measure, a decoding difficultymeasure that identifies an average phonetic similarity with other words,a weighted decoding difficulty measure that identifies decodingdifficulty weighted by frequency of words, a mean pronunciationvariation measure that identifies an average number of pronunciationvariations per word, a multiple pronunciation per word measure thatidentifies the proportion of words that contain multiple pronunciationvariations, a connected word per words measure that identifies theproportion of word sequences that are frequently pronounced as connectedwords, a co-articulated syllable measure that identifies a proportion ofco-articulated syllable pairs to the total number of syllables per word,a weak preposition per words measure that identifies the proportion ofprepositions used in weak form over the total number of words, acomplicated syllables measure that identifies the proportion ofcomplicated syllables over the total number of syllables, a word startwith weak syllable measure that identifies the proportion of wordsstarting with weak syllables (e.g., “assist”) over the total number ofwords, a median word frequency measure based on a spoken languagecorpus, a low frequency words measure that identifies the number ofunique words that appear rarely in spoken language as identified by acorpus, an idioms per clause metric, a phrasal verb per clause measure,a noun collocation per clause measure, a mean clause length measure, anumber of long sentences measure, a number of homophones measure, aphonetic neighbor density metric that identifies the number of neighborswhere neighbor is defined as a word that differs only in on phoneme, anda frequency weighted phonetic neighbor density measure. Certain of themeasures are pure acoustic based (wpsec) or pure text based(sent_per_1000words), while others are both acoustic and text based(phn_shift).

Upon selection of a collection of measures to be used, a scoring modelis calibrated to weight those measures accordingly. For example, a highspeaking rate measure (wpsec) may be positively weighted based on apositive correlation with speech sample difficulty (faster speakers areharder to understand), while a sentence length measure(sent_per_1000words) may be negatively weighted based on a negativecorrelation with speech sample difficulty (shorter sentences are easierto understand).

In addition to analyzing the difficulty of a speech sample that includesspeech of a single user, a speech sample scoring engine can beconfigured to analyze certain measures related to discourse betweenmultiple persons. For example, a speech sample may include aconversation between a man and a woman. The automatic speech recognizeror the speech sample scoring engine is configured to identify whichportions of the speech sample are associated with which speaker. Certainmeasures may be determined based on the characteristics of theindividual speaker. For example, certain pronunciation and prosodymeasures are determined for each speaker for consideration by thescoring model. Additionally, certain measures may be extracted based onthe interaction between the two speakers. For example, metrics may bedetermined that measure dialog characteristics of the communicationbetween the two speakers. Such dialog characteristics are converted intomeasures that are considered by the scoring model in generating adifficulty score for the scoring sample.

Difficulty scores associated with speech samples may be utilized in avariety of contexts. FIG. 4 is a block diagram depicting examplecontexts for usage of a difficulty score. A speech sample 402 isprovided to an automatic speech recognizer 404 for determination of wordhypotheses and time stamp associations 406. Those recognizer outputs 406are utilized by a speech sample scoring engine 408 in conjunction withthe speech sample 402 to generate a difficulty score 410 for the speechsample 402. The difficulty score associated with the speech sample 202is used to place the speech sample into appropriate data stores forsubsequent use.

For example, the speech sample 402 may be a studio recording of a textread aloud by a native English speaker. The speech sample 402 isgenerated for use as a teaching aid and for use as part of an item on anexamination. The speech sample 402 is provided to an automatic speechrecognizer having an acoustic model trained on native English speakersto generate word hypotheses, time stamp associations, and other acousticmeasures 406. The difficulty score 410 associated with the speech sample402 is used to appropriately classify the speech sample for anappropriate learning audience. For example, the difficulty score 410 mayrepresent a grade level for which the speech sample 402 is expected tobe understandable but challenging. Based on the difficulty score 410,the speech sample 402 can be put in an appropriate test bank 412 for usein examinations for students of the identified grade level.Additionally, the speech sample 402 can be put in a teaching aidrepository 414 for use in practice items for students of the identifiedgrade level.

In another example, the speech sample 402 is being analyzed forappropriateness for use in testing potential employees for a job. Thejob, such as a cook, may often involve listening to very fast,non-native English speakers, who do not use particularly difficultvocabulary. In such an example, the automatic speech recognizer 404 mayinclude an acoustic model trained using non-native speakers. Further,the scoring model of the speech sample scoring engine 408 may beconfigured to highly value speech samples 402 having a high speakingrate, while penalizing samples that use difficult vocabulary. Using suchlogic, an examination or training materials can be tailored to thescenario at hand, to automatically identify appropriate speech sampleswith minimal human intervention.

FIG. 5 is a flow diagram depicting a computer-implemented method ofassigning a difficulty score to a speech sample. Speech recognition isperformed on a digitized version of the speech sample using an acousticmodel to generate word hypotheses for the speech sample at 502. At 504,time alignment is performed between the speech sample and the wordhypotheses to associate the word hypotheses with corresponding sounds ofthe speech sample. A first difficulty measure is determined at 506 basedon the word hypotheses, and a second difficulty measure is determined at508 based on acoustic features of the speech sample. A difficulty scorefor the speech sample is generated at 510 based on the first difficultymeasure and the second difficulty measure.

Examples have been used to describe the invention herein, and the scopeof the invention may include other examples. FIGS. 6A, 6B, and 6C depictexample systems for use in implementing a speech sample scoring engine.For example, FIG. 6A depicts an exemplary system 600 that includes astandalone computer architecture where a processing system 602 (e.g.,one or more computer processors located in a given computer or inmultiple computers that may be separate and distinct from one another)includes a part of speech sample scoring engine 604 being executed onit. The processing system 602 has access to a computer-readable memory606 in addition to one or more data stores 608. The one or more datastores 608 may include speech samples 610 as well as difficulty scores612.

FIG. 6B depicts a system 620 that includes a client server architecture.One or more user PCs 622 access one or more servers 624 running a partof speech sample scoring engine 626 on a processing system 627 via oneor more networks 628. The one or more servers 624 may access a computerreadable memory 630 as well as one or more data stores 632. The one ormore data stores 632 may contain speech samples 634 as well asdifficulty scores 636.

FIG. 6C shows a block diagram of exemplary hardware for a standalonecomputer architecture 650, such as the architecture depicted in FIG. 6Athat may be used to contain and/or implement the program instructions ofsystem embodiments of the present invention. A bus 652 may serve as theinformation highway interconnecting the other illustrated components ofthe hardware. A processing system 654 labeled CPU (central processingunit) (e.g., one or more computer processors at a given computer or atmultiple computers), may perform calculations and logic operationsrequired to execute a program. A non-transitory processor-readablestorage medium, such as read only memory (ROM) 656 and random accessmemory (RAM) 658, may be in communication with the processing system 654and may contain one or more programming instructions for performing themethod of implementing a part of speech sample scoring engine.Optionally, program instructions may be stored on a non-transitorycomputer readable storage medium such as a magnetic disk, optical disk,recordable memory device, flash memory, or other physical storagemedium.

A disk controller 660 interfaces one or more optional disk drives to thesystem bus 652. These disk drives may be external or internal floppydisk drives such as 662, external or internal CD-ROM, CD-R, CD-RW or DVDdrives such as 664, or external or internal hard drives 666. Asindicated previously, these various disk drives and disk controllers areoptional devices.

Each of the element managers, real-time data buffer, conveyors, fileinput processor, database index shared access memory loader, referencedata buffer and data managers may include a software application storedin one or more of the disk drives connected to the disk controller 660,the ROM 656 and/or the RAM 658. Preferably, the processor 654 may accesseach component as required.

A display interface 668 may permit information from the bus 652 to bedisplayed on a display 670 in audio, graphic, or alphanumeric format.Communication with external devices may optionally occur using variouscommunication ports 672.

In addition to the standard computer-type components, the hardware mayalso include data input devices, such as a keyboard 673, or other inputdevice 674, such as a microphone, remote control, pointer, mouse and/orjoystick.

Additionally, the methods and systems described herein may beimplemented on many different types of processing devices by programcode comprising program instructions that are executable by the deviceprocessing subsystem. The software program instructions may includesource code, object code, machine code, or any other stored data that isoperable to cause a processing system to perform the methods andoperations described herein and may be provided in any suitable languagesuch as C, C++, JAVA, for example, or any other suitable programminglanguage. Other implementations may also be used, however, such asfirmware or even appropriately designed hardware configured to carry outthe methods and systems described herein.

The systems' and methods' data (e.g., associations, mappings, datainput, data output, intermediate data results, final data results, etc.)may be stored and implemented in one or more different types ofcomputer-implemented data stores, such as different types of storagedevices and programming constructs (e.g., RAM, ROM, Flash memory, flatfiles, databases, programming data structures, programming variables,IF-THEN (or similar type) statement constructs, etc.). It is noted thatdata structures describe formats for use in organizing and storing datain databases, programs, memory, or other computer-readable media for useby a computer program.

The computer components, software modules, functions, data stores anddata structures described herein may be connected directly or indirectlyto each other in order to allow the flow of data needed for theiroperations. It is also noted that a module or processor includes but isnot limited to a unit of code that performs a software operation, andcan be implemented for example as a subroutine unit of code, or as asoftware function unit of code, or as an object (as in anobject-oriented paradigm), or as an applet, or in a computer scriptlanguage, or as another type of computer code. The software componentsand/or functionality may be located on a single computer or distributedacross multiple computers depending upon the situation at hand.

It should be understood that as used in the description herein andthroughout the claims that follow, the meaning of “a,” “an,” and “the”includes plural reference unless the context clearly dictates otherwise.Also, as used in the description herein and throughout the claims thatfollow, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise. Further, as used in the description hereinand throughout the claims that follow, the meaning of “each” does notrequire “each and every” unless the context clearly dictates otherwise.Finally, as used in the description herein and throughout the claimsthat follow, the meanings of “and” and “or” include both the conjunctiveand disjunctive and may be used interchangeably unless the contextexpressly dictates otherwise; the phrase “exclusive or” may be used toindicate situation where only the disjunctive meaning may apply.

It is claimed:
 1. A computer-implemented method of delivering a speechsample to an appropriate audience based on a difficulty level of thespeech sample, comprising: retrieving the speech sample for evaluationfrom a non-transitory computer-readable memory; performing speechrecognition of the speech sample using a processing system to generateword hypotheses for the speech sample, the speech recognition using anautomatic speech recognizer configured using an acoustic model;performing, using the processing system, time alignment between thespeech sample and the word hypotheses to associate the word hypotheseswith corresponding sounds of the speech sample; determining, using atextual difficulty determination engine executed on the processingsystem, a first difficulty measure based on the word hypotheses for thespeech sample, the first difficulty measure estimating a level ofcontent difficulty of the speech sample; determining, using an acousticdifficulty determination engine executed on the processing system, asecond difficulty measure based on only acoustic features of the speechsample; generating, using the processing system, a difficulty scoreassociated with the speech sample by applying a scoring model to thefirst difficulty measure and the second difficulty measure; storing,using the processing system, the difficulty score associated with thespeech sample in a non-transitory computer-readable memory; selecting,based on the difficulty score, the speech sample associated with thedifficulty score for an appropriate audience; and outputting theselected speech sample to the appropriate audience.
 2. The method ofclaim 1, further comprising: determining, using the processing system, athird difficulty measure based on both acoustic features of the speechsample and the word hypotheses for the speech sample; wherein saidgenerating of the difficulty score further includes applying the scoringmodel to the third difficulty measure.
 3. The method of claim 2, whereinthe third difficulty measure is based on pronunciation quality in thespeech sample.
 4. The method of claim 2, further comprising: accessingan expected pronunciation for a word in the word hypotheses; comparingthe expected pronunciation to sounds of the speech sample associatedwith the word; determining a pronunciation acoustic feature based onsaid comparing as the third difficulty measure.
 5. The method of claim1, wherein the second difficulty measure is based on prosody of thespeech sample.
 6. The method of claim 1, wherein the first difficultymeasure is a vocabulary measure, a grammar measure, or a discoursemeasure.
 7. The method of claim 1, wherein the second difficulty measureis a fluency measure based on the number of words spoken per second, anumber of silences per word, a pause distribution, a speech chunklength, a speech chunk distribution, or a disfluency distribution. 8.The method of claim 1, wherein the second difficulty measure is aprosody measure based on a proportion of words starting with weaksyllables over a total number of words, a mean distance between stressedsyllables, an F0 range, an F0 slope, an F0 continuity, or an F0variation.
 9. The method of claim 1, wherein the first difficultymeasure is a vocabulary measure based on a comparison of the wordhypotheses with a word frequency table.
 10. The method of claim 9,wherein the word frequency table identifies a frequency of use ofdifferent words in one or more reference speeches or texts.
 11. Themethod of claim 1, further comprising: selecting the speech sample forinclusion on an examination based on the difficulty score.
 12. Themethod of claim 1, further comprising: selecting the speech sample foruse as a teaching aid based on the difficulty score.
 13. The method ofclaim 1, wherein the difficulty score indicates an education grade levelassociated with the speech sample.
 14. The method of claim 1, whereinthe difficulty score is generated based on a speaking rate measure, avocabulary difficulty measure, and a sentence length measure.
 15. Themethod of claim 1, wherein the speech sample is a speech sample from anon-native English speaker.
 16. The method of claim 15, wherein theautomatic speech recognition is based on an acoustic model trained usingnon-native English speakers.
 17. The method of claim 1, wherein thespeech sample is a speech sample from a native English speaker.
 18. Themethod of claim 17, wherein the automatic speech recognition is based onan acoustic model trained using native English speakers.
 19. Acomputer-implemented method of delivering a speech sample to anappropriate audience based on a difficulty level of the speech sample,comprising: retrieving the speech sample for evaluation from anon-transitory computer-readable memory, the speech sample includesspeech by multiple speakers; dividing the speech sample into multiplesegments according to a speaker identity; performing speech recognitionof the speech sample using a processing system to generate wordhypotheses for the speech sample, the speech recognition using anautomatic speech recognizer configured using an acoustic model;performing, using the processing system, time alignment between thespeech sample and the word hypotheses to associate the word hypotheseswith corresponding sounds of the speech sample; determining, using atextual difficulty determination engine executed on the processingsystem, a first difficulty measure based on the word hypotheses for thespeech sample, the first difficulty measure estimating a level ofcontent difficulty of the speech sample; determining, using an acousticdifficulty determination engine executed on the processing system, asecond difficulty measure based on only acoustic features of the speechsample; generating, using the processing system, a difficulty scoreassociated with the speech sample by applying a scoring model to thefirst difficulty measure and the second difficulty measure; wherein atleast one difficulty measure is determined for each of the multiplesegments; wherein the difficulty score is generated based on the atleast one difficulty measure determined for each of the multiplesegments; storing, using the processing system, the difficulty scoreassociated with the speech sample in a non-transitory computer-readablememory; selecting, based on the difficulty score, the speech sampleassociated with the difficulty score for an appropriate audience; andoutputting the selected speech sample to the appropriate audience. 20.The method of claim 19, wherein at least one difficulty measure isdetermined for an entirety of the speech sample; wherein the difficultyscore is generated further based on the at least one difficulty measurethat is determined for the entirety of the speech sample.
 21. The methodof claim 20, wherein a discourse measure is one of the at least onedifficulty measure that is determined for the entirety of the speechsample.
 22. A computer-implemented system for assigning delivering aspeech sample to an appropriate audience based on a difficulty level ofthe speech sample, comprising: a non-transitory computer-readable mediumconfigured to store a digitized version of a speech sample; an automaticspeech recognizer configured to perform speech recognition on thedigitized version of the speech sample using an acoustic model togenerate word hypotheses for the speech sample and to perform timealignment between the speech sample and the word hypotheses to associatethe word hypotheses with corresponding sounds of the speech sample; atextual difficulty determination engine configured to determine a firstdifficulty measure based on the word hypotheses for the speech sample,the first difficulty measure estimating a level of content difficulty ofthe speech sample; an acoustic difficulty determination engineconfigured to determine a second difficulty measure based on onlyacoustic features of the speech sample; a difficulty score calculatorconfigured to generate a difficulty score associated with the speechsample by applying a scoring model to the first difficulty measure andthe second difficulty measure; a computer-readable medium configured tostore the difficulty score associated with the speech sample; aselection module configured to select, based on the difficulty score,the speech sample associated with the difficulty score for anappropriate audience; and an output device configured to output theselected speech sample to the appropriate audience.